A Local-Global Approach to Semantic Segmentation in Aerial Images

Visual context is important in object recognition and it is still an open problem in computer vision. Along with the advent of deep convolutional neural networks, using contextual information with such systems starts to receive attention in the literature. At the same time, aerial imagery is gaining momentum. While advances in deep learning make good progress in aerial image analysis, this problem still poses many great challenges. Aerial images are often taken under poor lighting conditions and contain low resolution objects, many times occluded by trees or taller buildings. In this domain, in particular, visual context could be of great help, but there are still very few papers that consider context in aerial image understanding. In this thesis we introduce context as a complementary way of recognizing objects. We propose a dual stream deep neural network model that processes information along two independent pathways, one for local and another for global visual reasoning. The two are later combined in the final layers of processing. Our model learns to combine local object appearance as well as information from the larger scene in a complementary way, such that together they form a powerful classifier. We test our dual stream network on the task of segmentation of buildings and roads in aerial images and obtain state-of-the-art results on the Massachusetts Buildings Dataset. We also introduce two new datasets, for buildings and roads segmentation, respectively, and study the relative importance of local appearance versus the larger scene, as well as their performance in combination. We also extend the segmentation task to other classes that we find in aerial imagery, namely meadows, forest and water. While our local global model could also be useful in general recognition tasks, we clearly demonstrate the effectiveness of visual context in conjunction with deep nets for aerial image understanding.